1. Field of the Invention
The invention is generally related to digital image processing, and, in particular, is related to video stabilization.
2. Description of Related Art.
Video cameras are becoming more popular today as they become more widely available at lower prices. A video camera records sequential images within “frames.” A frame is a representation of an image at an instant of time. Each frame in a video segment (i.e., a sequence of frames) represents the image at a different instant in time. When several frames are recorded, at sequential instances in time, and are shown to the human eye in quick succession, the human eye is able to discern motion in the video segment.
Video (i.e., moving pictures) normally consist of a lot of motion, including object motion, such as a bird flying, and camera motion, such as camera panning, zooming, and tilting. The human eye may feel uncomfortable if the motion in the video is too fast, especially if the motion is changing very quickly in its direction or speed.
Other than natural motion of an object (e.g., one that is moving away from a stable or non-moving video camera), the video camera may record unwanted motion. The term “unwanted” usually refers to camera motion frequently resulting from hand jittering or from the camera person walking or otherwise moving. Other unwanted motion may result if, for example, wind shakes a mounted camera or a vehicle carrying a camera traverses rough terrain. Unwanted motion is very common in home videos.
Unwanted motion is also said to include “high frequency” components. The term “high frequency” refers to the motion changing very quickly in direction (e.g., left to right) or in speed (e.g., getting very fast or slowing down) in a unit of time (e.g., one second).
Video stabilization refers to a process for compensating for “unwanted” motion in a video segment. Video stabilization has been discussed in the literature for a number of years. The problem of “shaking shot” is common in home video. When a hand held camera is unsteady, video frames may be blurred or displaced, making viewing of the video difficult. Some high-end hand held video cameras support hardware video stabilization devices, notably “liquid lens” and electronic solutions. Some video cameras with such electronic solutions include the Sony® DCR-TRV900 video camera available from Sony Electronics, Inc. and the-Canon® ZR20 video camera available from Canon, U.S.A. Inc.
Such devices incorporating electronic solutions are expensive and are typically unavailable to the casual photographer. Moreover, these electronic solutions handle small amounts of movement. The electronic solutions are unable to distinguish between scanning and a lot of shaking, so they are unable to eliminate shaking. Overall, the electronic solutions try to slow down all motion in the video, without distinguishing between wanted and unwanted motion.
Additionally, video stabilization has been a topic in academic research. In some cases, the motion between each pair of consecutive frames or between each frame and a chosen origin frame is estimated. Based on the motion estimation, some inter-frame transformation parameters are computed for transforming (sometimes referred to as warping) each frame with respect to the origin frame. This technique, however, does not work when the camera is moving normally (e.g., panning, zooming, or tilting). Instead, the technique works only with a static camera.
Another conventional solution for unwanted motion is to map the estimated motion to some camera motion model (e.g., panning, zooming, tilting, etc.). Based on the camera model, each frame is transformed (i.e., its pixels are adjusted). There are several difficulties in these techniques. For example, motion estimation in video is time consuming and inaccurate when there are many outliers (i.e., data points that do not fall in the range of other data points). Another difficulty is that it is not uncommon to have several camera motions simultaneously, and, therefore, application of a single camera motion model leads to inaccurate results.